Stochastic Dynamic Programming Model Research Articles

Sustainable ecosystem management based on optimal water allocation and equity measures

Abstract Allocation of water resources is an interesting research topic and one of the main challenges of arid regions. From the point of view of agriculture, this issue is closely related to ecological balance, economic development and social stability. Therefore, fair, efficient and sustainable allocation of water resources for users and decision-making is essential. This paper presents a dynamic stochastic programming model that predicts soil moisture content in a growing season based on data collected from an experimental farm. The model included three types of loam soil, silt loam and clay loam with three treatments of irrigation intervals of 3, 7, 10 and 14 days and three amounts of water allocation with three replications. The proposed framework was evaluated with two criteria of spatial and temporal equity, and the optimal water allocation was analyzed based on this criterion. The results showed that the criterion of temporal equity for loam soil with 7-day irrigation intervals is more than twice that of 14-day irrigation intervals. In addition, the depth of irrigation has had the greatest impact on the fluctuations of the criterion of equity in water allocation in the growing season.

The small-bat-in-summer paradigm: Energetics and adaptive behavioural routines of bats investigated through a stochastic dynamic model.

Strong seasonality at high latitudes represents a major challenge for many endotherms as they must balance survival and reproduction in an environment that varies widely in food availability and temperature. To avoid energetic mismatches caused by limited foraging time and stochastic weather conditions, bats employ the energy-saving state of torpor during summer to save accumulated energy reserves. However, at high-latitude small-bats-in-summer face a particular challenge: as nocturnal foragers, they rely on the darkness at night to avoid predators and/or interspecific competition, but live in an environment with short, light summer nights, and even a lack of true night at the northernmost distributions of some bat species. To predict optimal behaviour in relation to latitudinal variation in diurnal cycles, we constructed a stochastic dynamic programming model of bats living at high latitudes. Using a stochastic dynamic programming framework with values that are representative for our study system, we show that individual energetic reserves are a strong driver of daytime use of torpor and night-time foraging behaviour alike, with these linked effects being both temperature- and photoperiod-dependent. We further used the model to predict survival probabilities at five locations across a latitudinal gradient (60.1° N to 70.9° N), finding that combinations of photoperiod and temperature conditions limited population distributions in the model. To verify our model results, we compared predictions for optimal decisions with our own empirical data collected on northern bats (Eptesicus nilssonii) from two latitudes in Norway. The similarities between our predictions and observations provide strong evidence that this model framework incorporates the most important drivers of diurnal decision-making in bat physiology and behaviour. Comparing empirical data and model predictions also revealed that bats facing lighter night conditions further north restrict their mass gain, which strengthens the hypothesis that predation threat is a main driver of bat nocturnality. Our model findings regarding state-dependent decisions in bats should contribute to the understanding of how bats cope with the summer challenges at high latitudes.

Vessel weather routing subject to sulfur emission regulation

The International Maritime Organization (IMO) rolled out a set of new regulations that limit the sulfur content in the fuel oil used by ships, which came into effect from 2020. This study aims to explore the operational decisions of vessels inside and outside emission control areas (ECAs) under uncertain weather conditions and scrutinizes the economics of two types of ships: fuel-switching and scrubber installation. Such operational decisions include speed, trim, and path decision. We construct a navigation grid system, in which each grid contains weather parameters and sea condition. This navigation grid system can be applied to other type of ships due to the well-established cost structure. To improve fuel consumption estimation, we train fuel consumption estimation functions with respect to weather parameters, sea conditions, and operational decisions. This study then develops a stochastic dynamic programming model to optimize the path decisions, vessel speed, and trim. To evaluate the efficiency of the developed stochastic dynamic programming model, we further conduct a case study on a Post-Panamax container vessel that operates westbound from Hamburg to Houston. We identify the path decision, speed and trim under ECAs and the uncertain weather conditions. Surprisingly, this case study demonstrates that the ship utilizing fuel-switching takes a detour trip; in contrast, the ship with a scrubber chooses the shortest path by maximizing the speed. Additionally, we observe a knock-on effect — scrubber installation lifts carbon dioxide emissions due to increased vessel speed outside the ECA region. Rich managerial insights and potential policy implications are discussed.

Reinforcement learning and stochastic dynamic programming for jointly scheduling jobs and preventive maintenance on a single machine to minimise earliness-tardiness

This paper addresses the problem of stochastic jointly scheduling of resumable jobs and preventive maintenance on a single machine, subject to random breakdowns, to minimise the earliness-tardiness cost. The main objective is to investigate using trending machine learning-based methods compared to stochastic optimisation approaches. We propose two different methods from both fields as we solve the same problem firstly with a stochastic dynamic programming model in an approximation way, then with an attention-based deep reinforcement learning model. We conduct a detailed experimental study according to solution quality, run time, and robustness to analyse their performances compared to those of an existing approach in the literature as a baseline. Both algorithms outperform the baseline. Moreover, the machine learning-based algorithm outperforms the stochastic dynamic programming-based heuristic as we report up to 30.5% saving in total cost, a reduction of computational time from 67 min to less than 1 s on big instances, and a better robustness. These facts highlight clearly its potential for solving such problems.

A stochastic linear programming model for maximizing generation and firm output at a reliability in long-term hydropower reservoir operation

The firm output, usually determined in the design stage of a hydroplant to serve as a threshold to measure system reliability, can be regarded as an unknown parameter to be explored to its maximum in long-term hydropower reservoir operation (LHRO). An unknown firm output to be ensured at certain reliability, however, will make the problem much more nonlinear and then complicate the modeling. This work presents a stochastic linear programming (SLP-1) model that can explicitly incorporate reliability in ensuring an unknown firm output to be maximized by using a probability variable to represent a decision at a state and introducing binary variables to decide whether the decision will ensure the unknown firm output. The present SLP-1 is improved on 1) another previous SLP model (SLP-0) that must have a firm output prespecified at certain reliability, and compared with 2) the stochastic dynamic programming (SDP) model that can only manage to estimate the firm output at a reliability with trial and error. Case studies show the superiority of the present SLP to the SDP that can hardly make the reliability any closer to what the SLP can achieve, especially in ensuring a high firm output, with gaps to desired reliability ranging up to 42.33% for Xiaowan and 31.8% for Nuozhadu. Indeed, the SLP-1 will encounter the dimensional difficulty that needs further efforts to overcome when applied to cascaded reservoirs.

Quantitative Analysis on Resilience of Distributed Energy Systems with Approximate Stochastic Dynamic Programming Models Considering Disaster Predictability

Quantitative Analysis on Resilience of Distributed Energy Systems with Approximate Stochastic Dynamic Programming Models Considering Disaster Predictability

Dynamic inventory decisions for humanitarian aid materials considering budget limitations

In case of disaster, providing relief supplies to the affected people has vital importance. Governmental or non-governmental organizations (NGOs) prepare for disasters by purchasing and stockpiling these aid materials in appropriate quantities. They operate under a limited budget and this budget can either be used before the disaster for stocking decisions under uncertainty or it can be used after the disaster to satisfy the required demand at a higher cost when the uncertainty is resolved. If all the budget is used before the disaster, and if there is no disaster in a long term, there will be a high holding cost. On the other hand, if all the budget is reserved for use after the disaster, meeting the demand will be more costly or the demand may not be met within a certain period of time. Thus, NGOs need to decide how to allocate the budget for pre and post-disaster usage. In this system, the budget of NGOs may also change over time through donations or other incomes. In this point of view, NGOs need to make dynamic stock and budget allocation decisions, under the available budget at hand. In our study, we analyze the dynamic stocking decisions of NGOs using stochastic dynamic programming formulations under budget constraints. We develop infinite horizon stochastic dynamic programming models with and without budget considerations, and compare the results of these models via numerical analysis. Detailed numerical studies and results of the sensitivity analysis show the significance of budget considerations in inventory decisions and the effects of different parameters on the system results.

Dual-Channel Retailer’s Multi-Period Inventory and Pricing Policy with the Quantity-Sales Mode

This paper investigates a dual-channel retailer’s inventory and pricing integrated decision-making problem over multiple periods. The quantity-sales mode is adopted in the online channel while the unit-sales mode is used offline. The customer demand of each sales channel is stochastic, and depends on visit costs, customer return costs, sales modes and pricing levels. To achieve the long-term profit maximization, the dual-channel retailer needs to optimize the pricing levels of both channels and the ordering quantity. For this problem, we first formulate a stochastic dynamic programming model, and then prove the joint concavity and the supermodularity of the objective function. With these properties, the optimality of the base-stock policy is proved, and the optimum price policy is characterized. Specifically, if the retailer’s optimum stock is neither too low nor too high, both sales channels need to be operated simultaneously, otherwise, only operating one channel is optimal. The online price is invariant with the retailer’s stock level while the offline price is decreasing in the retailer’s stock level. Finally, numerical results show that the quantity-sales mode, dual-channel sales model, and the multi-period dynamic decisions could benefit the dual-channel retailer. In addition, influences of cost parameters on the total profit are verified numerically.

Gray whale habitat use and reproductive success during seismic surveys near their feeding grounds: comparing state-dependent life history models and field data

We used a stochastic dynamic programming (SDP) model to quantify the consequences of disturbance on pregnant western gray whales during one foraging season. The SDP model has a firm basis in bioenergetics, but detailed knowledge of minimum reproductive length of females (Lmin) and the relationship between length and reproductive success (Rfit) was lacking. We varied model assumptions to determine their effects on predictions of habitat use, proportion of animals disturbed, reproductive success, and the effects of disturbance. Smaller Lmin values led to higher predicted nearshore habitat use. Changes in Lmin and Rfit had little effect on predictions of the effect of disturbance. Reproductive success increased with increased Lmin and with higher probability of reproductive success by length. Multiple seismic surveys were conducted in 2015 off the northeast coast of Sakhalin Island, with concomitant benthic prey surveys, photo-identification studies, and whale distribution sampling, thus providing a unique opportunity to compare output from SDP models with empirical observations. SDP model predictions of reproductive success and habitat use were similar with and without acoustic disturbance, and SDP predictions of reproductive success and large-scale habitat use were generally similar to values and trends in the data. However, empirical estimates of the proportion of pregnant females nearshore were much higher than SDP model predictions (a large effect, measured by Cohen’s d) during the first week, and the SDP model overestimated whale density in the south and underestimated density around the mouth of Piltun Bay. Such differences in nearshore habitat use would not affect SDP predictions of reproductive success or survival under the current seismic air gun disturbance scenario.

Comparison of dynamic and static pricing strategies in a dual-channel supply chain with inventory control

This paper studies the dynamic versus static pricing problem of a manufacturer–retailer dual-channel supply chain where demand is stochastic and price sensitive, and inventory can be replenished periodically. Four different pricing strategies, i.e., both members take dynamic pricing (DD strategy), retailer takes dynamic pricing while manufacturer takes static pricing (DS strategy), retailer takes static pricing while manufacturer takes dynamic pricing (SD strategy), and both members take static pricing (SS strategy), are considered. Under each of the pricing strategies, stochastic dynamic programming models are developed to determine the optimal pricing and inventory policy so that each member’s total expected discounted profit over the planning horizon is maximized. Numerical studies are carried out to compare the performance of different pricing strategies. Results show that: the optimal inventory policy under different pricing strategies is a base-stock type; under DD strategy, both members should reduce prices as long as one member’s initial inventory level is above its base stock level; under DS and SD strategies, the member adopting dynamic pricing should reduce price if its initial inventory level is above its base stock level; there exists a Nash pricing strategy which is affected by market parameters including demand variety, market size, channel preference, and price sensitivities.

Optimization of a domestic microgrid equipped with solar panel and battery: Model Predictive Control and Stochastic Dual Dynamic Programming approaches

In this study, a microgrid with storage (battery, hot water tank) and solar panel is considered. We benchmark two algorithms, MPC and SDDP, that yield online policies to manage the microgrid, and compare them with a rule based policy. Model Predictive Control (MPC) is a well-known algorithm which models the future uncertainties with a deterministic forecast. By contrast, Stochastic Dual Dynamic Programming (SDDP) models the future uncertainties as stagewise independent random variables with known probability distributions. We present a scheme, based on out-of-sample validation, to fairly compare the two online policies yielded by MPC and SDDP. Our numerical studies put to light that MPC and SDDP achieve significant gains compared to the rule based policy, and that SDDP overperforms MPC not only on average but on most of the out-of-sample assessment scenarios.

Optimal Bi-Level Bidding and Dispatching Strategy Between Active Distribution Network and Virtual Alliances Using Distributed Robust Multi-Agent Deep Reinforcement Learning

The deregulated active distribution network (ADN) would incorporate numerous autonomous stakeholders, including some emerging distributed virtual alliances (DVAs) like virtual microgrids and virtual power plants. Those DVAs would autonomously participate in energy market trading through bidding among themselves and dispatching conducted by the ADN. In this paper, the optimal bidding and dispatching model for DVAs and ADN in the day-ahead market is first developed as a stochastic dynamic programming model with the risk of misconduct considered, and then re-formulated as a Markov Decision Process to be solved by a new Distributed Robust Multi-Agent Deep Deterministic Policy Gradient algorithm based on the concept of robust Nash equilibrium (RNE). This algorithm is a fully distributed online optimization that would converge to RNE. It is an effective risk-averse method to obtain the optimal bidding strategies of DVAs and the optimal dispatching decisions of distribution system operator (DSO). Its high computational performance is demonstrated in the case studies, and the strategic decisions of DVAs and DSO are thoroughly analyzed.

A stochastic dynamic programming model for hydropower scheduling with state-dependent maximum discharge constraints

We present a medium-term hydropower scheduling model that includes inflow- and volume-dependent environmental constraints on maximum discharge. A stochastic dynamic programming algorithm (SDP) is formulated to enable an accurate representation of nonconvex relationships in the problem formulation of smaller hydropower systems. The model is used to assess the impact of including state-dependent constraints in the medium-term hydropower scheduling on the calculated water values. The model is applied in a case study of a Norwegian hydropower system with multiple reservoirs. We find that the maximum discharge constraint significantly impacts the water values and simulated operation of the hydropower system. A main finding is that the nonconvex characteristics of the environmental constraint are reflected in the water values, implying a nonconvex objective function. Operation according to the computed water values is simulated for cases with and without the environmental constraint. Even though operation of the system changes considerably when the environmental constraint is included, the total electricity generation over the year is kept constant, and the total loss in expected profit is limited to less than 0.8%.

Impacts of Yield and Seasonal Prices on the Operation of Lancang Cascaded Reservoirs

This work formulates a stochastic dynamic programming (SDP) model that incorporates seasonal electricity prices and can handle a constraint on power yield, which is assumed to be satisfied at any time it is possible, thus allowing for an analysis of their impacts on the operational performances of cascaded reservoirs. The model is applied to the Lancang Cascade, specifically its two largest reservoirs, Xiaowan and Nuozhadu. The results show that increasing the power yield of the cascade will reduce energy production unfavorably but will impact water spillage favorably, with a power yield of 2000 MW and with a 91% reliability suggested as being a satisfactory operational target. The case study also suggests that using seasonal electricity prices makes the power generation very unstable during weeks 12–20, which is a period of time that is critical to transferring from dry to flooding seasons.

Dynamic optimal control and simulation for unreliable manufacturing systems under perishable product and shelf life variability

This paper develops an optimal control policy for unreliable manufacturing systems evolving in a stochastic and dynamic environment and producing perishable products with limited and random shelf lives. The developed optimal control policy aims to minimize the total cost, composed of backlog, inventory holding and disposal costs. To do this, we develop a stochastic dynamic programming model and the obtained optimality conditions are solved using a numerical approach to determine the parametrized structure of the resulting optimal control policy. A simulation-based optimization approach is used to optimize the parameters of the developed control policy. This policy is subsequently analyzed using multiple sensitivity analyses and compared to other policies from the literature. We also study shelf life variability and its influence on the system performance. Numerical examples show that in the case of a random shelf life, the developed control policy presents a lower total cost in comparison with other policies from the literature.

Optimal Water Allocation Under Climate Change, Based on Stochastic Dynamic Programming Model Approach in Ribb Reservoir, Amhara, Ethiopia

Optimal Water Allocation Under Climate Change, Based on Stochastic Dynamic Programming Model Approach in Ribb Reservoir, Amhara, Ethiopia

A Stochastic Dynamic Programming Model for Hydropower Scheduling with State-Dependent Maximum Discharge Constraints

A Stochastic Dynamic Programming Model for Hydropower Scheduling with State-Dependent Maximum Discharge Constraints

PV-EV Integrated Home Energy Management Considering Residential Occupant Behaviors

Rooftop photovoltaics (PV) and electrical vehicles (EV) have become more economically viable to residential customers. Most existing home energy management systems (HEMS) only focus on the residential occupants’ thermal comfort in terms of indoor temperature and humidity while neglecting their other behaviors or concerns. This paper aims to integrate residential PV and EVs into the HEMS in an occupant-centric manner while taking into account the occupants’ thermal comfort, clothing behaviors, and concerns on the state-of-charge (SOC) of EVs. A stochastic adaptive dynamic programming (ADP) model was proposed to optimally determine the setpoints of heating, ventilation, air conditioning (HVAC), occupant’s clothing decisions, and the EV’s charge/discharge schedule while considering uncertainties in the outside temperature, PV generation, and EV’s arrival SOC. The nonlinear and nonconvex thermal comfort model, EV SOC concern model, and clothing behavior model were holistically embedded in the ADP-HEMS model. A model predictive control framework was further proposed to simulate a residential house under the time of use tariff, such that it continually updates with optimal appliance schedules decisions passed to the house model. Cosimulations were carried out to compare the proposed HEMS with a baseline model that represents the current operational practice. The result shows that the proposed HEMS can reduce the energy cost by 68.5% while retaining the most comfortable thermal level and negligible EV SOC concerns considering the occupant’s behaviors.

Predicting the population consequences of acoustic disturbance, with application to an endangered gray whale population.

Acoustic disturbance is a growing conservation concern for wildlife populations because it can elicit physiological and behavioral responses that can have cascading impacts on population dynamics. State-dependent behavioral and life history models implemented via Stochastic Dynamic Programming (SDP) provide a natural framework for quantifying biologically meaningful population changes resulting from disturbance by linking environment, physiology, and metrics of fitness. We developed an SDP model using the endangered western gray whale (Eschrichtius robustus) as a case study because they experience acoustic disturbance on their summer foraging grounds. We modeled the behavior and physiological dynamics of pregnant females as they arrived on the feeding grounds and predicted the probability of female and offspring survival, with and without acoustic disturbance and in the presence/absence of high prey availability. Upon arrival in mid-May, pregnant females initially exhibited relatively random behavior before they transitioned to intensive feeding that resulted in continual fat mass gain until departure. This shift in behavior co-occurred with a change in spatial distribution; early in the season, whales were more equally distributed among foraging areas with moderate to high energy availability, whereas by mid-July whales transitioned to predominate use of the location that had the highest energy availability. Exclusion from energy-rich offshore areas led to reproductive failure and in extreme cases, mortality of adult females that had lasting impacts on population dynamics. Simulated disturbances in nearshore foraging areas had little to no impact on female survival or reproductive success at the population level. At the individual level, the impact of disturbance was unequally distributed across females of different lengths, both with respect to the number of times an individual was disturbed and the impact of disturbance on vital rates. Our results highlight the susceptibility of large capital breeders to reductions in prey availability, and indicate that who, where, and when individuals are disturbed are likely to be important considerations when assessing the impacts of acoustic activities. This model provides a framework to inform planned acoustic disturbances and assess the effectiveness of mitigation strategies for large capital breeders.

Methodology of emergency medical logistics for multiple epidemic areas in public health emergency.

Coronavirus disease 2019(COVID-19) has brought great disasters to humanity, and its influence continues to intensify. In response to the public health emergencies, prompt relief supplies are key to reduce the damage. This paper presents a method of emergency medical logistics to quick response to emergency epidemics. The methodology includes two recursive mechanisms: (1) the time-varying forecasting of medical relief demand according to a modified susceptible-exposed-infected- Asymptomatic- recovered (SEIAR) epidemic diffusion model, (2) the relief supplies distribution based on a multi-objective dynamic stochastic programming model. Specially, the distribution model addresses a hypothetical network of emergency medical logistics with considering emergency medical reserve centers (EMRCs), epidemic areas and e-commerce warehousing centers as the rescue points. Numerical studies are conducted. The results show that with the cooperation of different epidemic areas and e-commerce warehousing centers, the total cost is 6% lower than without considering cooperation of different epidemic areas, and 9.7% lower than without considering cooperation of e-commerce warehousing centers. Particularly, the total cost is 20% lower than without considering any cooperation. This study demonstrates the importance of cooperation in epidemic prevention, and provides the government with a new idea of emergency relief supplies dispatching, that the rescue efficiency can be improved by mutual rescue between epidemic areas in public health emergency.